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Abstract 


Research  in  the  area  of  systems  engineering  competency  has  yielded  significant 
Department  of  Defense  (DoD)  policies,  guidance,  and  reference  materials  pertaining  to 
applying  systems  engineering  (SE)  policy,  standards,  and  best  practices  for  higher 
performance  on  major  acquisition  programs.  These  best  practices  are  not  universally 
applied  across  the  DoD  and  particularly  to  the  US  Army  Research,  Development,  and 
Engineering  Command  (RDECOM).  Further  research  is  needed  to  enable  acquisition, 
research,  and  development  efforts  with  the  ability  to  tailor  existing  documentation  and 
augment,  as  needed,  to  better  focus  the  breadth  and  depth  to  which  policies,  guidance, 
and  extent  of  documentation  is  utilized. 

This  research  task  assessed  the  current  state  of  systems  engineering  maturity  within 
RDECOM  to  benchmark  its  position  relative  to  its  practices  to  establish  a  plan/roadmap 
to  aid  RDECOM  in  creating  a  Systems  Engineering  Organization  Standard  Process 
appropriate  for  a  world  class  Research  &  Development  organization.  This  research  task 
utilized  an  RDECOM  analysis  of  its  SE  competencies  to  develop  appropriate  platforms 
to  fill  competency  gaps  and  deliver  appropriate  training  content  and  competency 
assessment  tools  to  provide  qualitative  and  quantitative  guidance  to  RDECOM  in  the 
development  of  a  SE  organizational  competency. 
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1  Introduction 


In  October  2004,  the  United  States  signed  into  law  the  Federal  Workforce  Flexibility  Act 
requiring  each  federal  agency  to  evaluate,  on  a  regular  basis,  its  training  programs  and 
plans  with  respect  to  the  accomplishments  of  its  specific  performance  plans  and 
strategic  goals.  This  law  was  in  part  a  response  to  the  need  for  advanced  human  capital 
with  improvements  in  labor  productivity  and  profitability  to  address  ever-increasing 
uses  of  new  and  advanced  technology  solutions.  Secondly,  this  law  was  responding  to 
the  large  deficit  in  technical  leadership  in  the  Federal  Government  (OPM  2009). 

The  challenges  for  developing  human  capital  with  a  high  technical  competency  while 
filling  a  technical  leadership  gap  have  not  been  confined  to  the  Federal  Government  or 
its  technical  domains.  Other  fields,  such  as  culinary  and  healthcare,  have  also  identified 
these  emerging  and  growing  issues  (Calhoun,  Dollett  et  al.  2008).  While  some  technical 
domains  have  established  legacy  in  the  definition  and  development  of  human  capital, 
e.g.  mathematics,  physics,  mechanical  engineering,  there  are  growing  fields  of  study  that 
have  become  critical  to  effective  and  efficient  technical  success  that  are  far  less  defined 
in  the  development  of  human  capital.  One  such  field  is  systems  engineering  (SE) 
(Shenhar  and  Sauser  2009),  which  has  only  recently  begun  to  better  define  how  the 
competency  in  this  field  is  developed  and  matured  (Squires  and  Larson  2009).  For  the 
advancement  and  development  of  human  capital  in  this  field,  the  Federal  Government 
and  its  agencies  will  need  to  expand  their  career  development  programs.  Therefore  in 
support  of  this  growth,  any  career  development  program  is  more  easily  integrated 
within  an  organization  when  it  is  based  on  competencies  needed  to  perform  the  job 
(Mirabile  1985). 

As  such  the  US  Army  Research  Development  and  Engineering  Command  (RDECOM) 
has  continued  to  advance  the  domain  and  practical  knowledge  of  systems  engineering  to 
optimize  their  ability  to  develop  systems  with  ever-increasing  environmental  and 
technical  complexities.  Therefore,  in  the  development  of  human  capital  for  the 
Department  of  Defense  (DoD)  and  RDECOM,  this  research  task  assessed  the  current 
state  of  SE  maturity  within  RDECOM  to  benchmark  its  position.  The  SE  knowledge  gaps 
present  will  be  utilized  to  establish  a  plan/roadmap  to  aid  RDECOM  in  establishing  a 
Systems  Engineering  Organization  Standard  Process  appropriate  for  a  world  class 
Research  &  Development  organization.  In  preparation  for  this  task,  RDECOM 
performed  a  gap  analysis  to  determine  the  competency  needs  SE.  This  task  utilized  this 
analysis  to  develop  appropriate  materials  to  fill  this  gap  and  deliver  appropriate  training 
content  and  competency  assessment  tools  to  provide  qualitative  and  quantitative 
guidance  to  RDECOM  in  the  development  of  a  SE  organizational  competency. 
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2  Approach 


In  the  creation  of  a  career  development  and  competency  model  for  an  organization,  it 
becomes  a  combination  of  observable  and  applied  knowledge,  skills  and  behaviors  and 
how  the  human  capital  ultimately  creates  value  to  what  is  actually  accomplished,  while 
focusing  on  the  behavior  rather  than  personality  traits  of  the  human  capital  (Jauhari 
and  Misra  2004).  Creating  competency-based  career  development  and  a  supporting 
model  consists  of  the  analysis,  assessment,  and  evaluation  of  necessary  job  skills.  This 
project  was  executed  within  three  phases  as  describe  in  Table  1.  The  results  of  this  three- 
phased  approach  was  a  SE  competency  development  and  assessment  platform  that  was 
used  to  train  and  assess  the  fundamental  knowledge  and  skills  of  a  RDECOM  SE 
workforce.  The  goal  of  this  work  is  to  advance  the  state  of  knowledge  and  practice  of  SE 
for  RDECOM. 


Table  1:  Project  Phases 


Phase 

Activity 

Outcomes 

Discovery 

Assessment  of  current  career  development  practices 
and  competency  models  in  SE  as  utilized  in 

RDECOM. 

Understanding  of  the  state  of  RDECOM 
needs  for  skill  levels  and  define  applicable 
competency  requirements  in  SE. 

Survey  current  RDECOM  practices  and  needs 
through  Voice  of  the  Customer  with  key  personnel 
and  the  RDECOM  SE  Integrate  Product  Team  (IPT). 

Analysis 

Analyze  gaps,  targets,  and  strategic  intent  based  on 
the  results  of  the  Discovery  phase  to  formulate  a 
position  for  executing  a  competency  development 
and  assessment  platform. 

Preliminary  version  of  a  SE  competency 
development  and  assessment  platform. 

Synthesis 

Execute  the  competency  development  and 
assessment  platform  via  a  spiral  development 
approach. 

Creation  of  a  SE  competency  development 
and  assessment  platform  for  further 
execution  within  RDECOM  and  throughout 
DoD  where  applicable. 
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3  Results 


Phase  1:  Discovery 

Phase  l  was  intended  to  determine  an  initial  set  of  competency  requirements  for 
RDECOM  SE.  These  requirements  would  be  based  on  organizational  needs,  gaps,  and 
future  developments  as  they  related  to  building  a  SE  knowledge  base  in  RDECOM.  The 
key  stakeholder  in  guiding  this  development  was  the  RDECOM  SE  Integrated  Product 
Team  (SE  IPT).  The  SE  IPT  consist  of  representatives  from  the  lead  organizations  within 
RDECOM  that  practice  SE  and  is  the  governing  body  of  all  SE  policy  and  practices 
within  RDECOM.  As  a  baseline,  the  SE  IPT  developed  an  initial  set  of  competency 
requirements  in  SE  that  were  necessary  for  developing  a  knowledge  base  in  SE  within 
RDECOM  (see  Appendix  A).  This  set  of  requirements  was  further  prioritized  into  a  set  of 
competency  modules.  While  these  modules  did  not  cover  the  entire  set  of  requirements 
or  SE  competencies,  they  were  defined  as  the  key  SE  competencies  of  greatest  priority  to 
RDECOM.  They  were: 

•  Modules  l:  Approaching  Problems 

o  Critical  vs.  Intuitive  Thinking 
o  Problem  Solving 
o  Systems  Thinking 

o  Soft  Systems  vs.  Hard  Systems  Methodology 
o  How  to  Model  Problems 

•  Module  2:  Engaging  Stakeholders 

o  Voice  of  the  Customer 

o  Perspectives:  Identifying  Potential  Customer  Voices 
o  Types  of  Requirements 
o  Functional  vs.  Non-Functional 
o  Capabilities  and  Characteristics 
o  Kano  Theory 
o  Listening  to  the  Customer 

o  Methods  for  Capturing  VoC  (Voice  of  the  Customer) 
o  QFD  (Quality  Function  Deployment)  and  House  of  Quality 

•  Module  3:  Defining  the  Solution  Space 

o  Pugh  Matrix 

o  Context  Diagrams  and  Scenario  Diagrams 
o  Developing  the  System  Requirements 
o  System  Architecture 
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Phase  2:  Analysis/Phase  3:  Synthesis 

Phase  2  transitioned  the  results  of  Phase  1  into  a  set  of  training  modules  supported  by 
proper  assessment  methods.  This  phase  had  three  developments:  training  objectives, 
training  materials,  individual  assessment  tool,  and  course  assessment  tool.  As  part  of 
Phase  3,  the  competency  development  and  assessment  platform  was  executed  via  a 
spiral  development  approach  that  allowed  for  refinement  and  assessment. 


Training  Objectives 

Based  on  the  three  modules  defined  from  Phase  l,  a  set  of  objectives  was  establishing 
from  initially  defined  content  and  topics.  These  objectives  were  written  in  the  context  of 
Bloom’s  Taxonomy  (Bloom  1956),  which  articulates  the  cognitive  domains  involved  in 
knowledge  and  the  development  of  intellectual  skills.  These  objectives  were  refined  as 
the  content  of  the  modules  evolved,  and  the  final  objectives  are  listed  in  Appendix  B. 


Training  Materials 

An  initial  set  of  training  materials  was  developed  based  on  the  outcomes  of  Phase  1  from 
pre-exiting  training  materials  developed  by  Stevens  Institute  of  Technology.  This 
content  was  refined  based  on  numerous  iterative  reviews  with  either  the  RDECOM 
Project  Headquarters  at  Aberdeen  Proving  Ground  or  the  RDECOM  SE ITP  (see  Figure 
1  for  a  timeline  of  this  review  process).  During  this  review  process  numerous 
refinements  were  made  before  the  first  deployment  of  the  materials  on  June  25-29,  2012 
at  the  US  Army  Aberdeen  Proving  Ground  (APG)  in  Aberdeen,  MD.  The  content  went 
through  further  refinement  after  the  APG  offering,  before  it  was  offered  at  the  US  Army 
Armament  Research  Development  and  Engineering  Center  (ARDEC)  at  Picatinny 
Arsenal,  NJ  on  July  16-19,  2012.  Each  course  finished  with  a  Technical  Review 
presentation  made  by  a  group  of  3-5  students.  For  the  Technical  Review,  they  were 
responsible  for  giving  a  20-minute  presentation  showing  their  analysis  of  the  in-class 
case  using  the  systems  engineering  artifacts  they  developed  and  refined  throughout  the 
week.  Presentations  were  then  evaluated  based  on  the  criteria  specified  in  Appendix  C. 
The  outline  of  the  presentations  was  as  follows: 

i)  Problem  Statement 

ii)  Systemigram 

iii)  Stakeholders  (Active  and  Passive) 

iv)  Stakeholder  Requirements  and  Priorities 

v)  Potential  System  Concepts 

vi)  Pugh  Matrix 

vii)  Context  Diagram 

viii)  Functional  Analysis 

ix)  QFD 

x)  System  Requirements  -  Functional 

xi)  System  Requirements  -  Input/Output 

xii)  Non-Functional  Requirements 
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xiii)  Functional  Architecture 

xiv)  Logical  to  Physical  Allocation 


Some  of  the  notable  changes  made  during  these  iterations  were: 

•  The  incorporation  of  content  being  developed  under  RT4  “Developing  SE 
Technical  Leadership”  (see:  http : / / www. sercuarc. or g /proi ects / view/ 6) . 

•  A  final  module  on  Project  Planning  was  replaced  with  a  module  on  Technical 
Reviews  and  the  students  would  perform  a  readiness  review  on  the  final  day  of 
training. 

•  The  reduction  of  the  class  from  a  5-day  class  to  a  3-day  class  to  better 
accommodate  the  loads  and  demands  of  the  RDECOM  personnel. 


The  final  outline  of  the  course  was  as  follows: 

Syllabus: 

Day  1:  Problems  with  Problems  /  Types  of  Stakeholders 

•  Welcome  and  Introduction 

•  General  Thinking  Overview 

•  Problem  Solving 

•  Problem  Definition 

•  Voice  of  the  Customer 
Day  2:  Engaging  Stakeholders 

•  QFD/House  of  Quality 

•  Context  Diagrams 

•  Pugh  Matrix 

•  Functional  Analysis 

Day  3:  Requirements  to  Architecture 

•  Developing  System  Requirements 

•  Functional  Architectures 

•  Physical  Architectures 

•  Technical  Baselines  and  Reviews 

•  Course  Evaluation 

•  Final  Exam 
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RDECOM 

Project 

Headquarters 

Review 

•  Jan.  10, 12,  25 


RDECOM  SE 
IPT  Review 

•  Jan.  31 


RDECOM 

Project 

Headquarters 

Review 

•  March  2,  9,  23, 
27,  30 


RDECOM  SE 
IPT  Review 

•  April  23-26 


Deployment  at 
Aberdeen 
Proving 
Ground 

•  June  25-29 


RDECOM  SE 
IPT  and 
Project 
Headquarters 
Review 

•  July  1-9 


Deployment  at 
ARDEC 

•  July  16-19 


Figure  l:  Development  Timeline 


Individual  Assessment  Tool 

One  of  the  requirements  established  by  the  SE  IPT  was  that  the  students  be  held 
accountable  for  the  materials  and  having  an  ability  to  effectively  assess  their 
competencies.  As  such  a  pre-  and  post-exam  was  developed  to  test  the  student’s  SE 
knowledge  prior  to  the  course  and  after  the  course.  Appendix  D  is  the  exam  with  the 
answers  and  question  weighting.  The  results  of  the  implementation  of  this  examination 
will  be  discussed  in  the  Assessment  section  of  this  report. 


Course  Assessment  Tool 

A  final  requirement  of  the  SE  IPT  was  the  ability  to  assess  the  course,  the  student’s 
views  of  the  course,  and  their  perception  of  the  course’s  ability  to  achieve  the  learning 
objectives.  This  assessment  tool  is  shown  in  Appendix  E. 


4  Assessment 


Fundamental  to  this  project  was  the  ability  to  assess  the  course,  instructors,  learning 
outcomes,  and  student’s  knowledge/performance  of  SE.  The  following  is  a  summary  of 
those  assessments. 
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Course  Assessment 

Aberdeen  Proving  Ground,  June  25-29 

The  following  is  the  executive  summary  provided  to  RDECOM  on  the  course 
assessment.  A  detailed  analysis  and  student  feedback  is  in  Appendix  F: 

The  course  went  well.  It  was  completed  comfortably  in  four  days  instead 
of  the  allotted  five  days  without  rushing  the  lectures  or  exercises.  Some 
positive  observations  that  may  be  gathered  from  the  comments  on  the 
student  assessments: 

•  Overall,  the  material  was  well  received  by  the  students 

•  The  students  are  hungry  for  tools  to  help  them  in  the  systems 
engineering  process 

•  They  seem  to  appreciate  approaches  to  formalize  the  systems 
engineering  process 

•  The  format  of  the  class  -  lecture  then  do  -  was  well  received 

•  The  more  significant  recommendation  for  course  improvement 
was  related  to  the  examples  in  the  lecture  material.  The  example 
that  runs  through  the  course  material  isAmazon.com.  The 
students  overwhelming  believe  the  example  should  be  one  from 
RDECOM/ Army  so  that  it  is  more  relevant  to  them. 

•  There  was  a  number  of  requests  for  similar  training  on  the  “right 
hand”  side  of  the  Systems  engineering  “V” 


Finally,  some  observations  from  the  professor’s  perspective: 

•  While  it  is  understood  why  the  IPT  decided  to  take  the  approach  on 
the  RFI  problem  utilized  for  the  class  exercise,  the  lack  of 
information  was  a  negative.  Few  of  the  students  had  any  idea 
what  the  RFI  was  requesting  -  it  was  just  too  sketchy  to  work 
from,  and  that  uncertainty  got  in  the  way  of  learning. 


ARDEC,  July  1-9 

The  following  is  the  executive  summary  provided  to  RDECOM  on  the  course 
assessment.  A  detailed  analysis  and  student  feedback  is  in  Appendix  G: 

The  course  went  well.  It  was  completed  comfortably  in  three  days.  The 
pace  was  comfortable.  Positive  observations  that  may  be  gathered  from 
the  comments  on  the  student  assessments: 

•  Overall,  the  material  was  well  received  by  the  students 
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•  The  students  liked  the  requirements  writing  material,  and  the  QFD 
material 

•  Class  exercises  were  very  helpful 

•  There  was  mixed  feedback  on  Systemigrams  this  time.  Some  liked 
the  approach,  some  thought  they  were  too  informal. 

•  There  is  stronq  interest  in  a  similar  course  to  address  the  right  side 
of  the  “V”. 

•  A  contrast  to  the  first  offering  at  APG,  is  that  this  group  were 
much  more  comfortable  with  the  in-class  robot  deployment  system 
-  and  the  lack  of  definition  of  the  problem. 


Student  Assessment 

Students  received  a  pre-examination  within  ten  minutes  of  arriving  to  the  class  and  had 
30  minutes  to  complete  the  exam.  At  the  end  of  the  course,  they  were  asked  to  repeat  the 
exam  with  30  minutes  to  complete  the  exam.  Results  of  the  pre-  and  post-exams  are 
shown  in  Tables  2  and  3.  All  exams  (APG  and  ARDEC)  were  graded  by  the  same  faculty 
member  to  assure  consistency  in  grading.  In  summary,  the  following  is  how  the  exams 
were  graded: 

[1]  A  5-point  scale  (1-5)  was  used  for  each  question.  If  the  student  did  not  answer 
the  question,  they  received  a  o.  If  the  question  had  multiple  parts,  the  points 
were  distributed  equally  across  the  parts. 

[2]  The  pre-exams  were  graded  first  and  then  the  post-exams.  The  student’s  pre¬ 
exam  was  not  used  as  a  baseline  or  comparison  for  determining  their  post-exam 
grade  on  any  one  question.  Pre-  and  Post-exams  were  graded  independently. 

[3]  Scores  per  question  were  input  into  an  Excel  spreadsheet  as  raw  data.  These 
scores  were  then  evaluated  using  the  weighting  (WT)  specified  per  question. 

[4]  Final  scores  were  reported  on  a  0-100  scale. 


Some  observations  about  the  pre-  and  post-examinations,  APG,  and  ARDEC: 

•  Some  students  did  not  preform  as  well  on  the  post-exam  on  a  few  questions. 
While  it  is  not  clear  why,  it  was  the  interpretation  of  the  grader  that  some 
students  may  have  given  up  and  wanted  to  finish  the  exam  without  properly 
answering  the  question. 

•  The  APG  class  used  an  open  book  format  for  the  post-exam,  while  the  ARDEC 
class  did  not.  It  should  be  expect  that  the  APG  post-exams  would  have  a  higher 
score  than  the  ARDEC  post  exams. 

o  APG  class  average  score:  82  (15) 
o  ARDEC  class  average  score:  74  (14) 
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•  The  ARDEC  personnel  have  received  more  fundamental  SE  training  coming  into 
the  class,  and  would  have  better  pre-exam  scores. 

o  APG  class  average  score:  33  (12) 
o  ARDEC  class  average  score:  53  (20) 

•  Based  on  the  previous  two  observations,  one  would  expect  the  delta  between  pre- 
and  post-exams  to  be  greater  for  the  APG  class  than  the  ARDEC  class. 

o  APG  class  delta  score:  48  (18) 
o  ARDEC  class  delta  score:  21  (14) 


Table  2:  APG  Pre-  and  Post-Examination  Grades 


Statistics 

Pre-Exam  Score 
(0-100) 

Post-Exam  Score 
(0-100) 

Exam  Score  Increase 

Pre  to  Post 

Mean  (sd) 

33(12) 

82(15) 

48  (18) 

Mode 

42 

98 

Median 

35 

85 

Max 

60 

98 

73 

Min 

12 

40 

4 

Table  3:  ARDEC  Pre-  and  Post-Examination  Grades 


Statistics 

Pre-Exam  Score 
(0-100) 

Post-Exam  Score 
(0-100) 

Exam  Score  Increase 

Pre  to  Post 

Mean  (sd) 

53 (20) 

74 (14) 

21 (14) 

Mode 

37 

94 

Median 

63 

73 

Max 

77 

94 

56 

Min 

6 

41 

2 

5  Future  Developments 


The  results  of  this  research  task  were  to  help  start  a  SE  competency  development  and 
assessment  platform  for  the  RDECOM  workforce.  This  was  based  on  a  set  of 
competency  requirements  in  SE  as  defined  by  RDECOM  (Appendix  A).  Addressing  all  of 
these  requirements  was  out  of  the  scope  of  this  research  task,  so  the  focus  was  on  the 
Technical  Processes  that  comprise  “the  left  side  of  the  ‘V’”  (see  Figure  2).  Further 
advancing  this  research  task  would  be  to  develop  the  appropriate  platforms  to  address 
the  “right  side  of  the  ‘V’”  and  the  corresponding  Technical  Management  Processes. 
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Figure  2:  Systems  Engineering  “V” 


It  would  also  be  recommended  to: 

•  Further  evaluate  the  current  body  of  knowledge  in  career  development  and 
competency  models  in  industry  and  other  government  agencies. 

•  Re-evaluate  key  competencies  developed  for  RDECOM  and  consider  broaden  key 
parameters  to  include  SE  proficiencies  needed  to  support  workloads  outside 
(other  than)  RDECOM. 

•  Assess  the  entire  DoD  population  of  SE  for  existing  proficiency  levels  and 
competency  needs. 

•  Develop  and  initiate  a  development  process  development  for  certification 
approvals  in  SE. 

•  Develop  SE  certification  levels/strata,  competency  definitions/elements  based  on 
population  assessment  and  execute  a  validation  effort  for  certification  assessment 
tools  and  assignment  and  certification  approvals. 

•  Address  SE  Behaviors  as  part  of  the  competency  development  and  assessment 
plan.  The  most  effective  competency  models  consider  the  behavioral  traits 
(Ryschkewitsch,  Schaible  et  al.  2009)  as  a  direct  correlation  to  optimal 
competency  performance. 

•  Expand  the  SE  knowledge  interchange  to  incorporate  multiple  platforms  and 
medium.  Building  SE  competencies,  behaviors,  and  skill  levels  for  the  RDECOM 
SE  workforce  will  take  a  comprehensive  and  systematic  knowledge  interchange 
approach  using  an  appropriate  mixture  of  standard  and  advanced  media.  Thus,  a 
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review  of  new  and  emerging  knowledge  interchange  media  (e.g.  social  media) 
could  be  assessed  to  determine  how  it  can  be  effectively  used  with  or  to  enhance 
current  medium.  The  appropriate  media  could  be  matched  with  the  Behaviors 
and  Competencies,  so  as  to  develop  a  systematic  SE  career  development  plan  that 
is  the  most  effective.  In  the  identification  of  appropriate  media  indicators,  key 
performance  criteria  could  be  considered  such  as  creativity,  innovation, 
communication,  collaboration,  critical  thinking,  and  technology.  See  Table  4  for 
an  example  of  this  multi-platform  knowledge  model. 


Table  4:  Systems  Engineering  Knowledge  Interchange  Media 


Knowledge 

Interchange 

Platforms 

Delivery  Method 

Duration 

Credit  Type 

Learning  Type 

Training  l 
(Short-range) 

Classroom 

Online 

Seminar 

Conference/Prof.  Meeting 

1-3  days 

Continuing  Ed. 
Certification 
(not  for  credit) 

Task-based 

or 

Content-based 

Training  2 
(Mid-range) 

Classroom 

Online 

4+  days 

Continuing  Ed. 
Certification 
(not  for  credit) 

Task-based 

or 

Content-based 

College  Course 

Classroom 

Online 

40+ 

contact 

hrs. 

College  (credit) 

Content-based 

Advanced  Degree 

Classroom 

Online 

12+ 

months 

Certificate 

Masters 

Ph.D. 

Content-based 

Personal 

Experience 

Rotation 

On-the-Job 

Participant  Observation 
Experience  Accelerator 

3+  days 
to  4+ 
months 

None 

Task-based 

Figure  3  represents  a  Career  Development  and  Competency  Model  (CDCM) 
Architecture  that  can  be  used  bring  the  SE  Competencies,  Behaviors,  and  Knowledge 
Interchange  Media  into  a  comprehensive  and  systemic  plan  that  can  not  only  build,  but 
sustain,  the  careers  of  Systems  Engineers  in  RDECOM.  The  development  and  execution 
of  this  plan  will  require  more  than  standard  practices  and  approaches,  but  must  look  for 
novel  approaches  to  developing  Systems  Engineers.  RDECOM  is  addressing  today’s  and 
tomorrow’s  challenge,  so  its  Systems  Engineers  will  have  to  gain  standard  and  emerging 
knowledge  in  SE.  Important  to  this  is  the  integration  of  Behaviors  with  Competencies 
while  building  and  transferring  these  through  Knowledge  Interchange  Media.  Figure  3 
is  a  context- architecture  of  the  development  of  the  CDCM  Plan.  The  end  objective  of  this 
would  be  to  have  an  executable  plan  leading  to  defined  Skill  Levels  and  organizational 
Certification. 
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Systems 

Engineering 

Knowledge 

Interchange 

Media 


Systems  Engineering 
Behaviors 


Systems  Engineering 
Competencies 


Systems 

Engineering 

Career 

Development 

and 

Competency 
Model  Plan 


Figure  3:  Career  Development  and  Competency  Model  Architecture 
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Appendices 


Appendix  A:  Initial  RDECOM  SE  Competency  Requirements 


Topic  Area 

Requirement 

Introduction 

Course  Objectives:  State  WHY  are  you  getting  trained  with  RDECOM  focus 

Purpose:  Train  RDECOM  SE  best  practices/process  areas,  technical  reviews. 
Facilitate  Communication/Integration  across  RDECOM 

RDECOMs  Role  in  the  Army  and  Impact  throughout  the  program  lifecycle: 

Develop  the  enabling  technology  for  PoRs,  Acquisitions  Matrix  Support,  Materiel 
Solution  Provider  and  Technical  Advisors,  Addressing  O&S  Cost  Drivers 

Overview  of  current  SE  initiatives;  WSARA,  Better  Buying  Power 

Overview  how  SE  is  executed  on  Joint  Programs  within  RDECOM  (Cross- 
Command) 

SE  Roles  &  Responsibilities  (part  of  OPORD  Overview) 

System 

Thinking 

System  Concept  overview 

Explain  the  SE  Linkages  Products/artifacts/processes  to  program  objectives  and 
decision  points 

Aspects  of  Development  that  need  to  be  address  across  the  Lifecycle  Phases; 

System  Under  Development,  M&S,  Test  Infrastructure,  Development 
Infrastructure/Environment  (SW  development  environment) 

System  Architecture  Overview 

CONOPs,  How  is  the  system  employed 

Engineering  Resilient  Systems 

Project 

Management 

Areas 

Project  Planning  Essentials 

Project  Plan  &  Quarterly  Review  Process  Outlined 

Team  Building  -  Teaming  Structure 

SE  Process 

Areas  - 

Illustrate 
inputs/ outputs/ 
activities/ 
recommended 
tools/ artifacts 

Running  Application/Example  - 

More  depth  in  how  to  execute  SE  Process  areas  -  include  MOEs/MOPs 

Flow-down  &  Use 

Stakeholder  Requirements  Definition 

Requirements  Analysis 

Architectural  Design 

Implementation 

Integration 

Verification 
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Lifecycle  Cost  Estimations  (LCCE) 

Peer  Review 

Validation 

Tech  Planning 

Technical  Assessment 

Configuration  Management 

Data  Management 

Requirements  Management 

Risk  Management 

Interface  Management 

Transition 

Decision  Analysis  and  Resolutions  (DAR) 

Tools 

SE  Tools  Overview 

DashBoard  Overview 

MBSE  Overview 
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Appendix  B:  Course  Description  and  Learning  Objectives 

Course  Description: 

Research  on  advances  in  systems  engineering  has  yielded  significant  DoD  policies,  guidance,  and 
reference  materials  pertaining  to  applying  systems  engineering  policy,  standards,  and  best  practices  for 
higher  level  TRL/Major  Acquisition  programs.  A  closer  review  of  Lower  TRL  research  and  development 
efforts  reveals  that  this  area  is  in  an  immature  state  in  terms  of  identifying  and  providing  applicable 
policy,  standards,  and  best  practices  to  assist  these  efforts.  This  course  will  help  aspiring  technical  leaders 
develop  and  refine  their  skills  in  analyzing  systems,  synthesizing  holistic  solutions,  and  making  sound 
judgments  in  the  presence  of  ambiguity,  rapid  change,  and  non-technical  constraints.  The  course  will  be  a 
mixed  model  delivery  of  lectures,  case  studies,  perspectives,  and  experiential  learning. 

Day  l  Learning  Objectives 


Objective 


Translate  and  Explain  system  thinking  objectives  into  a  problem  statement  that  can  be  solved  by 
traditional  engineering  techniques  and  skill  sets. 

Diagram  and  Discriminate  the  environment ,  system  under  development ,  and  suitability  of  a  systems 
solution  using  systems  thinking  approach. 

Apply  systems  thinking  tools  and  techniques  to  better  understand  the  enterprise  and  technology  issues , 
which  will  affect  the  design  of  a  system  and  translates  these  into  system  requirements 

Day  2  Learning  Objectives 


Objective 


Evaluate  and  Describe  customer  needs ,  active  and  passive  stakeholders  and  their  perceptions  of  a 
system  of  interest. 

Identify  essential  features  that  are  critical  to  satisfy  customers  and  Distinguish  where  to  focus 
further  improvement  efforts. 

Create  a  set  of  artifacts  (i.e.  Context  Diagram,  QFD,  Pugh  Matrix)  that  identify  key  drivers  of  customer 
satisfaction  and  needs. 

Day  3  Learning  Objectives 


Objective 


Demonstrate  the  concept  of  Use  Case  Scenarios  to  better  understand  the  “ Capabilities ”  that  the 
stakeholders  require. 

Translate  the  output  from  the  development  of  use  case  scenarios  and  system  objectives  into  system 
requirements  and  Critique  the  characteristics  offgood’frequirements. 

Comprehend  the  importance  and  the  role  of  the  system  architecture  and  associated  derived 
requirements  in  the  development  and  project  planning  process. 

Assess  a  system  of  interest  using  the  concepts  and  artifacts  learned  in  class  and  Arrange  this  in  a 
coherent  and  effective  Review. 
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Appendix  C:  Final  Project  Specifications 

Project  Problem: 

Technical  Reviews  assure  that  the  systems  development  and  technical  baselines  are  within  the  system 
objectives  to  satisfy  the  customer.  A  Technical  Review  provides  management  decision  points  where  trade¬ 
offs  can  be  exercised  to  maintain  convergence  on  a  system  solution  within  the  constraint  boundaries.  Your 
team  will  be  responsible  for  executing  a  Technical  Review  presentation  of  your  proposed  system  solution 
based  on  the  RFI  provided  in  Day  l  to  ensure  that  the  resulting  set  of  requirements  agrees  with  customer 
needs  and  expectations  and  that  the  system  under  review  can  proceed. 

Original  RFI:  541st  EN  CO,  54th  EN  BN  RCP  Soldiers  request  a  method  to  deploy  and  operate  their  robots 
without  exiting  their  vehicles 

Request:  Develop  and  integrate  an  exterior  mounted  RDS  with  a  low  profile. 

What  is  Due: 

Your  team  will  be  responsible  for  giving  a  20  minute  presentation  showing  your  analysis  of  the  case  using 
the  artifacts  you  developed  and  refined  throughout  the  week.  Your  presentation  should  include  the 
following  artifacts  articulated  through  the  tools  and  techniques  learned  in  class: 

1.  mission  objectives 

2.  candidate  system  architecture 

3.  top-level  system  requirements 

4.  operational  interfaces 

5.  preferred  system  concept 

6.  enabling  technologies 

Your  presentation  will  be  assessed  upon: 

1.  System  objectives  are  clearly  defined  and  stated  and  are  unambiguous. 

2.  The  preliminary  set  of  requirements  satisfactorily  provides  a  system  that  will  meet  the  objectives. 

3.  The  system  is  feasible.  A  solution  has  been  identified  that  is  technically  feasible. 

4.  The  concept  evaluation  criteria  to  be  used  in  candidate  systems  evaluation  have  been  identified  and 
prioritized. 

5.  The  presentation  utilized  the  learned  artifacts  to  demonstrate  the  learning  objectives. 


Evaluation  criteria: 


Rubric: 

0 

1 

2 

3 

4 

Technical  Requirements 

Did  it  include  the  artifacts  articulated  through  the  tools  and 
techniques  learned  in  class? 

Analysis  and  Synthesis 

Was  information  effectively  synthesized  from  the  tools  and 
techniques  learned  in  class  to  perform  an  analysis  of  the  data  or 
information. 

Presentation  and  Writing 

Did  it  flow  well?  Did  it  have  logical  progression? 

Was  information  clearly  presented?  Was  it  easy  to  understand? 
Were  graphs,  tables,  and  visuals  used? 

Did  it  have  organization  and  clarity 
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Appendix  D:  Pre-  and  Post- Examination 


l.  What  is  the  difference  between  analytical  and  intuitive  thinking?  Which  is  better  in  systems 
engineering? 

Weight:  10% 

Intuitive  thinking  relies  upon  context,  ambient  conditions,  modular  responsivity,  task  difficulty,  task 
ambiguity,  and/or  affective  state. 

Analytical  thinking  relies  upon  education,  training,  critical  thinking,  logical  competence,  rationality, 
feedback,  and/or  intellectual  ability. 


2.  Define  the  concept  of  “stakeholders.”  Describe  the  difference  between  active  and  passive  stakeholders. 

Weight:  5% 

Active:  Individuals,  Entities,  Other  Systems,  which  will  actively  interact  with  the  “system”  once  it  is 
operational  and  in  use 

Passive:  Individuals,  Entities,  Other  Systems,  Standards,  Protocols,  Procedures,  Regulations,  which  will 
also  influence  the  “success”  of  the  system 


3.  Choose  a  system  from  your  domain  and  identify  the  list  of  active  and  passive  stakeholders  for  this 
system. 

Weight:  5% 

List  of  Active  and  passive,  following  rules  from  question  2 


4.  Develop  a  context  diagram  for  the  above  system.  What  are  the  three  things  that  you  think  get  clarified 
and  communicated  by  a  context  diagram? 
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5.  What  is  a  stakeholder  requirement?  What  is  the  difference  between  a  stakeholder  capability 
requirement  and  a  stakeholder  characteristic  requirement?  Give  three  examples  of  each  from  the 
system  that  you  described  in  Question  3. 

Weight:  15% 

Capabilities: 

1)  Be  able  to  add,  change  and  delete  products  for  sale  at  any  time 

2)  Be  able  to  change  product  prices  at  any  time 

3)  Provide  secure  user  ID  and  password 

Characteristics: 

1)  Use  current  interface  defined  by  the  financial  institution 

2)  Don’t  use  too  much  bandwidth 

3)  Be  operational  within  6  months 


6.  What  is  a  system  requirement?  What  is  the  difference  between  a  system  functional  requirement  and  a 
system  non-functional  requirement?  Give  three  examples  of  each  from  the  system  that  you  described 
in  Question  3. 

Weight:  15% 

MIL-STD  499B  [Military  Standard,  1993]:  identifies  the  accomplishment  levels  needed  to  achieve  specific 
objectives. 

Chambers  and  Manos  [1992]:  the  attributes  of  the  final  design  that  must  be  a  part  of  any  acceptable 
solution  to  the  design  problem. 

Davis  [1993]:  a  user  need  or  necessary  feature,  function,  or  attribute  of  a  system  that  can  be  sensed  from  a 
position  external  to  that  system. 

Grady  [1993]:  an  essential  attribute  for  a  system  or  an  element  of  a  system,  coupled  by  a  relation 
statement  with  value  and  units  information  for  the  attribute. 

Harwell  et  al.  [1993]:  “If  it  mandates  that  something  must  be  accomplished,  transformed,  produced,  or 
provided,  it  is  a  requirement  -  period.” 
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7.  Give  six  attributes  of  a  good  system  requirement. 

Weight:  10% 

1.  Unambiguous 

2.  Understandable 

3.  Correct 

4.  Concise 

5.  Traced 

6.  Traceable 

7.  Design  independent 

8.  Verifiable 

9.  Unique 

10.  Complete 

11.  Consistent 

12.  Comparable 

13.  Modifiable 

14.  Attainable 


8.  Describe  the  objective  of  a  System  Requirements  Review  (SRR). 

Weight:  10% 

Stakeholders  identified,  Operational  requirements  understood,  draft  spec  created,  draft  performance 
defined,  top  level  functional  analysis  completed,  draft  SEP,  draft  TEMP,  etc. 


9.  What  is  a  system  functional  or  logical  architecture? 

Weight:  10% 

An  architecture  that  defines  what  the  system  must  do  -  a  partitioning  of  the  system’s  top  level  function 
into  sub-functions  that  describe  the  functionalities,  the  tasks,  and/or  the  processing  that  the  solution 
must  perform. 

This  limited  view  of  the  functional  architecture  is  the  most  common  and  is  represented  as  a  directed  tree. 
Or, 

A  model  that  augments  the  functional  partitioning  by  capturing  the  transformation  of  inputs  into  outputs 
using  control  information.  This  definition  adds  the  flow  of  inputs  and  outputs  throughout  the  functional 
partitioning. 

These  items  that  comprise  the  inputs  and  outputs  are  commonly  modeled  via  a  data  model. 
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Appendix  E:  Course  Evaluation  Form 

RDECOM  Advanced  SE  Training  Assessment 


Part  is  Course  Evaluation 


Please  rate  the  degree  to  which  you  agree 
with  the  following  statements. 

Strongly 
Disagree  -  0 

Disagree  - 1 

Neutral  -  2 

Agree  -  3 

Strongly 
Agree  -  4 

1 .  The  course  met  all  of  its  stated 
objectives. 

□ 

□ 

□ 

□ 

□ 

2.  The  course  was  relevant  to  my  job. 

□ 

□ 

□ 

□ 

□ 

3.  Overall,  the  course  materials  added  value 
to  my  learning  experience. 

□ 

□ 

□ 

□ 

□ 

4.  My  knowledge/  skills  increased  as  a 
result  of  this  course. 

□ 

□ 

□ 

□ 

□ 

5.  1  am  confident  that  1  am  able  to  apply 
knowledge/  perform  skills  gained. 

□ 

□ 

□ 

□ 

□ 

6.  Overall,  the  course  met  my  needs  and 
expectations. 

□ 

□ 

□ 

□ 

□ 

Part  2:  Instructor  Evaluation 


Please  rate  the  degree  to  which  you  agree 
with  the  following  statements. 

Strongly 
Disagree  -  0 

Disagree  - 1 

Neutral  -  2 

Agree  -  3 

Strongly 
Agree  -  4 

1 .  The  instructor  related  course  to 
participants’  needs  and  experiences. 

□ 

□ 

□ 

□ 

□ 

2.  The  instructor  promoted  participation 
discussion  and  engagement. 

□ 

□ 

□ 

□ 

□ 

3.  The  instructor  kept  the  discussion  on  topic 
and  the  activities  on  track. 

□ 

□ 

□ 

□ 

□ 

4.  Overall  the  instructor  was  effective. 

□ 

□ 

□ 

□ 

□ 

Part  3:  Participant  Comments 


Questions  J  Ifesppnse 


1 .  What  did  you  learn  that  will  be  most 
valuable  to  your  job? 

2.  What  methods  or  techniques  contributed 
to  your  learning/  engaged  you  most? 

3.  What  actions  might  you  or  others  take  to 
continue  to  build  these  knowledge/  skills? 

4.  What  suggestions  do  you  have  to 
improve  the  program? 

5.  What  topic  areas  would  attract  you  to 
future  programs? 

6.  Additional  Comments 
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Part  4:  Learning  Objectives 


Please  rate  the  degree  to  which  you  agree 
with  the  following  statements. 

No  New 
Learning  -  0 

Little  New 
Learning  - 1 

Some  New 
Learning  -  2 

Significant 
Learning  -  3 

Great 

Learning  -  4 

1 .  Translate  and  Explain  system  thinking 
objectives  into  a  problem  statement 
that  can  be  solved  by  traditional 
engineering  techniques  and  skill  sets. 

□ 

□ 

□ 

□ 

□ 

2.  Diagram  and  Discriminate  the 
environment,  system  under 
development,  and  suitability  of  a 
systems  solution  using  systems 
thinking  approach. 

□ 

□ 

□ 

□ 

□ 

3.  Apply  systems  thinking  tools  and 
techniques  to  better  understand  the 
enterprise  and  technology  issues, 
which  will  affect  the  design  of  a  system 
and  translates  these  into  system 
requirements. 

□ 

□ 

□ 

□ 

□ 

4.  Evaluate  and  Describe  customer 
needs,  active  and  passive 
stakeholders  and  their  perceptions  of  a 
system  of  interest. 

□ 

□ 

□ 

□ 

□ 

5.  Identify  essential  features  that  are 
critical  to  satisfy  customers  and 
Distinguish  where  to  focus  further 
improvement  efforts. 

□ 

□ 

□ 

□ 

□ 

6.  Create  a  set  of  artifacts  (i.e.  Context 
Diagram,  QFD,  Pugh  Matrix)  that 
identify  key  drivers  of  customer 
satisfaction  and  needs. 

□ 

□ 

□ 

□ 

□ 

7.  Demonstrate  the  concept  of  Use  Case 
Scenarios  to  better  understand  the 
“Capabilities”  that  the  stakeholders 
require. 

□ 

□ 

□ 

□ 

□ 

8.  T ranslate  the  output  from  the 
development  of  use  case  scenarios 
and  system  objectives  into  system 
requirements  and  Critique  the 
characteristics  of  “good”  requirements. 

□ 

□ 

□ 

□ 

□ 

9.  Comprehend  the  importance  and  the 
role  of  the  system  architecture  and 
associated  derived  requirements  in  the 
development  and  project  planning 
process. 

□ 

□ 

□ 

□ 

□ 

10.  Assess  a  system  of  interest  using  the 
concepts  and  artifacts  learned  in  class 
and  Arrange  this  in  a  coherent  and 
effective  Review. 

□ 

□ 

□ 

□ 

□ 
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Appendix  F:  Course  Evaluation  for  APG 

RT29  —  RDECOM  Advanced  SE  Course  Evaluation 

Taught  6/25/2012  -  6/28/2012  at  APG 


RDECOM  Advance  SE  Course.  June  25-2S.  2012 
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Figure  4:  Summary  of  Student  Assessment  Questions  (APG) 

There  were  6  questions  about  the  overall  course  content  (C-i  through  C-6). 


Table  5:  Overall  Course  Content  (APG) 


C-1 

The  course  met  all  of  its  stated  objectives 

C-2 

The  course  was  relevant  to  my  job 

C-3 

Overall,  the  course  materials  added  value  to  my  learning  experience 

C-4 

My  knowledge/skills  increased  as  a  result  of  this  course 

C-5 

1  am  confident  that  1  am  able  to  apply  knowledge/perform  skills  gained 

C-6 

Overall,  the  course  met  my  needs  and  expectations 
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There  were  10  questions  about  the  Learning  Objectives  for  this  course  (L-i  through  L-io). 
Table  6:  How  well  the  course  satisfied  the  Learning  Objectives  (APG) 


L-1 

Translate  and  Explain  system  thinking  objectives  into  a  problem  statement  that  can 
be  solved  by  traditional  engineering  techniques  and  skill  sets. 

L-2 

Diagram  and  Discriminate  the  environment,  system  under  development,  and 
suitability  of  a  systems  solution  using  systems  thinking  approach. 

L-3 

Apply  systems  thinking  tools  and  techniques  to  better  understand  the  enterprise  and 
technology  issues,  which  will  affect  the  design  of  a  system  and  translates  these  into 
system  requirements. 

L-4 

Evaluate  and  Describe  customer  needs,  active  and  passive  stakeholders  and  their 
perceptions  of  a  system  of  interest. 

L-5 

Identify  essential  features  that  are  critical  to  satisfy  customers  and  Distinguish 
where  to  focus  further  improvement  efforts. 

L-6 

Create  a  set  of  artifacts  (i.e.  Context  Diagram,  QFD,  Pugh  Matrix)  that  identify  key 
drivers  of  customer  satisfaction  and  needs. 

L-7 

Demonstrate  the  concept  of  Use  Case  Scenarios  to  better  understand  the 
“Capabilities”  that  the  stakeholders  require. 

L-8 

Translate  the  output  from  the  development  of  use  case  scenarios  and  system 
objectives  into  system  requirements  and  Critique  the  characteristics  of  “good” 
requirements. 

L-9 

Comprehend  the  importance  and  the  role  of  the  system  architecture  and  associated 
derived  requirements  in  the  development  and  project  planning  process. 

L-10 

Assess  a  system  of  interest  using  the  concepts  and  artifacts  learned  in  class  and 
arrange  this  in  a  coherent  and  effective  Review. 

There  were  4  questions  about  the  Instructor  (I-i  through  I-4). 


Table  7:  Instructor  Effectiveness  (APG) 


1-1 

The  instructor  related  course  to  participant's  needs  and  experience 

1-2 

The  instructor  promoted  participation  discussion  and  engagement 

1-3 

The  instructor  kept  the  discussion  on  topic  and  the  activities  on  track 

1-4 

Overall,  the  instructor  was  effective 

Figures  2  and  3  represent  the  average  scores  for  each  question  asked,  and  the  percent  of 
students  that  answered  “strongly  agree”  to  the  questions  asked. 
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Figure  5:  Average  Scores  on  Student  Assessment  Questions  (APG) 
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Figure  6:  Percent  Strongly  Agree  on  Student  Assessment  Questions  (APG) 
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Section  2:  This  section  contains  all  of  the  qualitative  comments  found  in  Part  3  (Participation 
Comments)  of  the  SE  Training  Assessment 

1.  What  did  you  learn  that  will  be  most  valuable  to  your  job? 

•  Methods  and  tools  which  could  be  applied  immediately  to  job. 

•  The  details  of  QFD 

•  Methods  for  capturing  stakeholder  needs/wants 

•  Methods  for  Functional  Decomposition 

•  Context  diagram 

•  Considering  systems/things  as  stakeholders 

•  The  “V”  model  and  use  the  technique  “what”  to  collect/generate  requirements 

•  The  multiple  tools  discussed  in  class.  Capturing  the  Voice  of  the  customer. 

•  I  won’t  have  cause  to  apply  any  of  this  in  my  job 

•  System  engineering  principles 

•  Various  tools  and  feedback  from  other  students  on  their  thoughts 

•  Breakdown  process  into  logical  steps 

•  Technical  review,  QFD,  Functional  architecture 

•  System  engineering  principles  &  tools 

•  Requirements  definition  process 

•  QFD  and  functional  architecture  tools 

•  How  to  perform  requirements  analysis  and  how  requirements  relate  to  overall  systems 
architecture 

•  Rigorous  process  for  identifying  problem  and  turning  that  into  requirements,  system 
concepts,  and  functional  architecture 

•  Different  tools  and  definitions  I  can  apply  to  my  job 

•  Coordination  w/  Stakeholders  value 

•  The  set  of  artifacts  (context  diagrams,  QFD  and  Systemigrams)  that  can  be  used  to  describe 
customer  needs. 

2.  What  methods  or  techniques  contributed  to  your  learning/  engaged  you  most? 

•  The  team  assignment 

•  Instructor  lead  student  discussions 

•  Details  on  System  Architecture  Development 

•  Actively  going  through  examples  as  a  class  (whiteboarding) 

•  Group  example  helpful  to  familiarize  with  tools  -  better  to  work  with  example  relevant  to  me 

•  Systemigram,  functional  vs.  non-functional,  QFD,  and  Pugh  matrix 

•  Instructor  sharing  past  experience  and  good  dialogue  from  fellow  co-workers 

•  Participation  group  assignments 

•  Exercise 

•  QFD 

•  QFD  and  functional  architecture  tools 

•  The  thinking  exercises  and  team  work  initiatives 

•  Exercises  based  on  materials 

•  The  instructor’s  experience,  knowledge,  room  presence,  and  enthusiasm. 

•  Instructor  was  knowledgeable,  and  could  convey  the  message  clearly 

•  Participating  in  the  exercise  immediately  after  the  lecture  was  presented. 

3.  What  actions  might  you  or  others  take  to  continue  to  build  these  knowledge/  skills? 

•  Use  it  on  the  job 

•  Read  the  reference  books  in  the  handouts 

•  Work  to  include  these  methods  on  my  project  as  we  are  in  the  requirements  phase  of  the  V. 

•  Follow-on  topic  areas  (maybe  V2  day  courses)  -  ex.  Architecture 
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•  Reading  materials/group  discussions 

•  Apply  directly  to  my  project  as  much  as  I  can 

•  Apply  tools  and  techniques  to  current  project  and  future  projects.  Internet  searches  to  learn 
additional  tools 

•  Attend  conferences  and  do  independent  research  regarding  specific  tools  of  interest 

•  Use  this  for  use  in  reviews 

•  Related  books  and  projects 

•  Institute  on  current  project 

•  Apply  tools  to  programs  that  I’m  working  on 

•  Apply  tools  to  day  to  day  activities  as  appropriate  upon  returning  to  office 

•  Apply  these  concepts  to  real  project.  When  students  are  assigned,  they  should  have  a  real 
need  for  something  in  the  class  and  management  support/mandate  to  actually  implement 

•  Research  tools  and  techniques  provided  by  the  instructor  that  will  increase  my  SE  knowledge 

•  This  course  was  an  overview  (high  level).  Maybe  break  up  course  for  deeper  understanding  of 
each  segment 

•  Apply  the  tools  immediately  to  current  and  upcoming  projects 

4.  What  suggestions  do  you  have  to  improve  the  program? 

•  Use  examples  of  RDECOM’s  ATO’s,  TECD  development  project 

•  The  role  of  1)  Strategy,  2)  Joint  Modeling  and  Simulation,  3)  Computerized  Scenarios 

•  Change  example  from  Amazon.com  to  something  that  includes  all  disciplines 

•  Change  the  RDS  or  add  to  it  for  more  data  to  help 

•  QFD  tool  is  out  of  date.  QFD  examples  are  unclear 

•  Define  example  better  for  group  exercise.  Make  more  relevant  to  audience. 

•  Include  Object  oriented  architecture  vs.  functional.  Work  example  of  both. 

•  Introduce  Doors  &  Rhapsody  tools 

•  Directly  using  DoD  example  instead  of  Amazon.com 

•  Do  a  case  study  of  a  DoD  project. 

•  Examples  more  relevant  to  RDECOM  developed  hardware  &  software 

•  Single  work  instead  of  group.  Get  a  better  QFD  tool.  Stick  w/  Amazon  or  commercial 
example.  DoD  ones  won’t  work.  They’re  too  complicated.  Uninstall  Systemitool  and  use  visio. 
Requirements  traceability  SW  use  would  be  useful  to  teach.  Use  a  PC  based  classroom  like 
6008.  Provide  area  map  of  food  to  off-base  attendees.  They’re  your  guests  &  spent  all  week 
eating  at  Burger  King. 

•  If  you  insist  on  group  work,  it  needs  to  be  a)  smaller  groups  &  b)  bigger  monitors 

•  Relate  more  directly  to  CERDEC  policies 

•  Forget  the  final  presentation,  waste  of  time.  Learning  was  in  group  versus  presentations 

•  A  real  problem  to  solution  instead  of  bits  and  pieces 

•  Do  not  use  Amazon.com  example 

•  Exercise  project  could  be  more  complex  and  encourage  more  creativity 

•  Sample  RFI  was  too  vague  -  need  to  select  an  RDECOM  program. 

•  Use  examples  from  real  DoD  programs  instead  of  commercial  projects  (Amazon.com) 

•  Examples  of  artifacts  to  be  based  on  Army  DoD  systems  and  not  commercial  products 

•  Don’t  have  exercise  time  flow  into  breaks,  people  tend  to  keep  going  if  they  don’t  have  hard 
deadline.  Do  AAR  immediately  after  exercise,  not  after  a  break  (like  lunch) 

•  More  time  with  the  tools.  Different  tools  besides  the  QFD 

•  Better  source  material.  Unreadable,  small  print,  incomplete 

•  Prior  to  beginning  the  course,  have  students  select  a  project  from  their  domain  and  use  it 
during  the  course 

5.  What  topic  areas  would  attract  you  to  future  programs? 

•  All 
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•  l)  Strategy,  2)  Modeling  and  Simulation,  3)  Computerized  Scenario  Development,  4) 
Orthogonal  “mathematical”  methods  of  best  system  selection 

•  More  on  Systems 

•  New  tools,  deep  dives  (focused).  Discussion  opportunities  -  Good,  bad,  ugly 

•  System  architectures 

•  Emerging  SE  tools  and  best  practices 

•  How  to  make  your  organization  use  this 

•  QFD 

•  Modular  Open  System  Approach  (MOSA)  and  DoD  Architecture  Framework  (DoDAF) 

•  Completing  the  second  leg  of  the  “V”  model 

•  The  other  side  of  the  “V”  implementation  -  Transition 

•  Tools  to  help  the  SE  process.  The  Integration  side  of  the  SE  Process 

•  See  below 

•  A  detailed  course  on  the  QFD  process. 

6.  Additional  Comments 

•  Print  course  material  in  one  chart  per  page,  not  two  per  page 

•  Would  like  to  see  course  on  tools  that  our  community  accepts.  Systemigrams  work,  but  not  in 
embedded  into  Industry  as  a  standard 

•  Provide  more  space  for  comments  on  Assessment  form 

•  Discussions  should  ask  for  good,  bad,  ugly  from  participants 

•  Questions  [on  exam]  don’t  align  to  course  (ex:  no  QFD  questions  or  Systemigram) 

•  Examples  are  not  easy  to  read  or  follow 

•  Provide  tool  kit  that  can  be  used  (preferable  to  book  of  slides)  -  Quick  Reference 

•  Develop  or  offer  systems  architecture  and  integration  follow-ons 

•  Compositional  SE  would  be  interesting  topic  to  do  more  with 

•  Familiarize  everyone  with  software/IA/Security 

•  Did  not  address  suitability 

•  Overall  the  course  is  excellent.  I  learned  a  lot  even  though  it  is  too  much  material  for  5  days 
course. 

•  Material  should  include  previous  effort  examples  from  my  agency 

•  Instructor  was  great  -  he  sometimes  was  off  topic  &  tangents,  but  interesting  and  necessary 

•  Don’t  force  specific  tools,  offer  options 

•  Too  much  material  to  cover  -  expand  it  into  two  sessions 

•  I  suggest  spending  more  time  on  developing  a  functional  architecture 

•  Suggest  that  different  project  problems  be  assigned  to  different  teams  as  a  diversity  when 
presentations  are  performed 

•  Keep  Amazon.com  example.  It’s  good  universal  example.  99%  of  people  are  familiar  with.  It’s 
a  counterbalance  to  running  project,  which  is  Army  specific. 

•  You  can  keep  RFI  vague  but  the  instructor  should  know  more  and  class  should  do  VOC  to 
elicit  details.  Answers  should  be  kept  in  shared  repository  (e.g.  whiteboard,  shared  drive,  etc) 

•  Some  items  in  SE  should  be  skipped  because  it  has  been  covered  in  DAU 

•  Engineers  should  be  reminded  that  it  is  the  lesson  to  learn  not  get  bogged  down  with  minor 
design  details 

•  The  testing  aspect  of  the  course  is  unnecessary.  Class  exercises/interactive  and  evaluation  of 
the  project  is  sufficient.  Materials  were  absolutely  horrible.  Only  one  approach  (and  NOT 
EVEN  the  favored  approach)  was  presented  in  the  material.  That  cancels  out  the  value  of 
taking  the  class.  Still,  there  was  learning  that  occurred. 

•  Good  course.  Learned  some  new  concepts  and  was  refreshed  on  concepts  that  I  learned  in 
previous  courses 
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Appendix  G:  Course  Evaluation  for  ARDEC 


RDECOM  Advanced  SE,  July  16-18,  Picatinny 
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C-1  C-2  C-3  C-4  C-5  C-6  L-1  L-2  L-3  L-4  L-5  L-6  L-7  L-8  L-9  L-10  I- 

Question  Number 


1  1-2  1-3  1-4 


■  Strongly  Agree/Great  Learning 
□  Neutral/Some  New  Learning 

■  Strongly  Disagree/No  New  Learning 


□  Agree/Significant  Learning 
■  Disagree/Little  New  Learning 


Figure  7:  Student  Assessments  (ARDEC) 


There  were  6  questions  about  the  overall  course  content  (C-i  through  C-6). 

Table  8:  Overall  Course  Content  (ARDEC) 


C-i 

The  course  met  all  of  its  stated  objectives 

C-2 

The  course  was  relevant  to  my  job 

C-3 

Overall,  the  course  materials  added  value  to  my  learning  experience 

C-4 

My  knowledge/skills  increased  as  a  result  of  this  course 

C-5 

I  am  confident  that  I  am  able  to  apply  knowledge/perform  skills  gained 

C-6 

Overall,  the  course  met  my  needs  and  expectations 

There  were  10  questions  about  the  Learning  Objectives  for  this  course  (L-i 
through  L-10). 
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Table  9:  How  well  the  course  satisfied  the  Learning  Objectives  (ARDEC) 


L-i 

1.  Translate  and  Explain  system  thinking  objectives  into  a  problem  statement 
that  can  be  solved  by  traditional  engineering  techniques  and  skill  sets. 

L-2 

2.  Diagram  and  Discriminate  the  environment,  system  under  development, 
and  suitability  of  a  systems  solution  using  systems  thinking  approach. 

L-3 

3.  Apply  systems  thinking  tools  and  techniques  to  better  understand  the 
enterprise  and  technology  issues,  which  will  affect  the  design  of  a  system  and 
translates  these  into  system  requirements. 

L-4 

4.  Evaluate  and  Describe  customer  needs,  active  and  passive  stakeholders 
and  their  perceptions  of  a  system  of  interest. 

L-5 

5.  Identify  essential  features  that  are  critical  to  satisfy  customers  and 
Distinguish  where  to  focus  further  improvement  efforts. 

L-6 

6.  Create  a  set  of  artifacts  (i.e.  Context  Diagram,  QFD,  Pugh  Matrix)  that 
identify  key  drivers  of  customer  satisfaction  and  needs. 

L-7 

7.  Demonstrate  the  concept  of  Use  Case  Scenarios  to  better  understand  the 
“Capabilities”  that  the  stakeholders  require. 

L-8 

8.  Translate  the  output  from  the  development  of  use  case  scenarios  and 
system  objectives  into  system  requirements  and  Critique  the  characteristics 
of  “good”  requirements. 

L-9 

9.  Comprehend  the  importance  and  the  role  of  the  system  architecture  and 
associated  derived  requirements  in  the  development  and  project  planning 
process. 

L-io 

10.  Assess  a  system  of  interest  using  the  concepts  and  artifacts  learned  in 
class  and  arrange  this  in  a  coherent  and  effective  Review. 

There  were  4  questions  about  the  Instructor  (I-i  through  I-4). 


Table  10:  Instructor  Effectiveness  (ARDEC) 


I-i 

The  instructor  related  course  to  participant's  needs  and  experience 

1-2 

The  instructor  promoted  participation  discussion  and  engagement 

1-3 

The  instructor  kept  the  discussion  on  topic  and  the  activities  on  track 

1-4 

Overall,  the  instructor  was  effective 

Figures  2  and  3  represent  the  average  scores  for  each  question  asked,  and  the  percent  of 
students  that  answered  “strongly  agree”  to  the  questions  asked. 
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Average  Score 


Question  Number 

Figure  8:  Average  Scores  on  Student  Assessment  Questions  (ARDEC) 

Percent  Strongly  Agree 
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Figure  9:  Percent  Strongly  Agree  on  Student  Assessment  Questions  (ARDEC) 
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Section  2:  This  section  contains  all  of  the  qualitative  comments  found  in  Part  3 
(Participation  Comments)  of  the  SE  Training  Assessment 

1.  What  did  you  learn  that  will  be  most  valuable  to  your  job? 

•  Functional  Analysis,  System  Analysis 

•  Reinforced  front  end  of  the  SE  “V”,  will  be  able  to  review  our  MBSE  methodology 

•  Effective  elicitation  of  stakeholder  rqts 

•  Pugh  matrix  for  decision  making 

•  Steps  of  SE 

•  Systemigrams,  QFD 

•  Writing  good  requirements 

•  Refresh  of  SE  practices  and  practiced  portions  of  SE  process  that  I  haven’t  had  experience 
with  yet  (i.e.  functional  analysis) 

•  Practical  group  project  examples,  Req.  Analysis  process 

•  The  general  concepts  in  the  3  parts  of  SE  top  down  approach  (SRD,  RA,  Arch) 

•  Writing  good  requirements 

•  Writing  requirements 

•  Structured  way  from  start  to  finish,  exercise 

•  A  thorough  review  of  QFD  and  House  of  Quality 

•  Additional  uses  of  QFD  (i.e.  find  missing  reqs) 

•  Creating  good  requirements 

2.  What  methods  or  techniques  contributed  to  your  learning/  engaged 
you  most? 

•  Systemigram,  QFD 

•  Activity  being  team  based 

•  Exercises 

•  Exercises  helped  reinforce  learning 

•  Group  exercise  with  practical  problems 

•  QFD 

•  Functional  analysis,  requirement  development 

•  Systemigram  development 

•  The  sections  where  we  executed  SE  in  teams  on  our  own 

•  On  hand  development  of  services 

•  Systemigram  was  good,  context  diagram,  have  seen  others 

•  Systemigram,  QFD 

•  The  examples  that  were  used  and  teamwork 

•  Class  participation 

•  Methods  of  collecting/organizing  req  &  the  tools  used 

3.  What  actions  might  you  or  others  take  to  continue  to  build  these 
knowledge/  skills? 

•  Keep  going  back  and  looking  at  material 

•  Be  sure  to  apply  to  projects  or  skills  will  be  lost 

•  Review  and  use  training  material 

•  Exercise  tools  on  projects 

•  Practice  on  real  tools 

•  Re-read  DAG  chapter  4 

•  Apply  to  the  projects 

•  Increasing  MBSE  skill  to  apply  all  of  these  practices 
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•  Pilot  on  project,  utilize  these  tools  and  see  if  there  is  value  in  actually  understanding  the 
problem  better 

•  Apply  the  knowledge  gained  and  review  material  taught 

•  Continue  to  practice 

•  Share  knowledge  within  program  IPT 

•  Use  class  material  as  reference  as  needed 

•  Put  it  into  play...  and  practice 

4.  What  suggestions  do  you  have  to  improve  the  program? 

•  Add  some  videos 

•  Dedicate  more  time  to  the  front  modules  1  -3 

•  Assign  exercises  to  each  student  on  team 

•  More  time  on  Systemigrams 

•  It  only  dealt  with  the  left  side  of  the  “V” 

•  A  little  more  time  on  tech  reviews  and  specifics  about  preparation  for  review  (i.e.  artifacts 
required 

•  Offer  a  V2  day  concentration  of  project  type  (TechD,  RDP  or  PM)  focused  training  does  in  real 
life 

•  Need  to  refine  purpose  of  Systemigrams 

•  I  think  it  was  too  fast  n  DASG  for  firsties 

•  Large  charts,  e.g.  systemigram,  should  not  be  shrunken  down  too  much 

•  Improve  Pre-V  presentations 

•  Nothing  really,  I  think  it  is  fine  that  we  work  on  the  left  side  of  the  “V” 

•  All  instructors  were  not  present  at  beginning,  certain  admin  remarks  were  duplicated 

•  N/A 

5.  What  topic  areas  would  attract  you  to  future  programs? 

•  Systemigram 

•  MBSE  enabled,  completing  the  “V” 

•  Systems  architecture  and  systems  thinking 

•  Functional  architecture 

•  QFD 

•  Should  be  followed  with  class  for  the  right  side  of  “V” 

•  The  right  side  of  the  “V”  should  be  taught  including  TRL  and  MRL  assessment 

•  MBSE  practices  (applying  these  skills) 

•  Example  of  successful  TechD  &  templates 

•  The  other  SE  side  of  the  “V” 

•  Right  side  of  “V”,  and  all  lifecycle  phases 

•  MBSE 

•  Architecture 

6.  Additional  Comments 

•  Overall,  very  well  prepped,  constructed,  and  delivered 

•  SEAA  (3-4  day  course)  should  be  a  pre-requisite  to  this  advanced  course.  SEAA  provides 
basic  essential  information  on  the  ARDEC  SE  process  and  procedure  but  doesn’t  provide 
details  on  how  to  use  tools. 

•  Working  on  the  class  exercise  with  hands-on  tools  was  great  to  grasp  better  understanding  of 
how  things  should  be  done. 

•  Systemigram  was  a  brand  new  concept  which  was  never  taught  before  or  mentioned  in  any 
previous  SE  teaching.  It  was  very  confusing.  It’s  too  much  in  a  free-form  style.  To  be  a  useful 
tool  there  should  be  formalism  put  into  the  way  it  can  be  used.  Having  essential  objects 
(boxes)  to  be  filled  out  can  be  more  beneficial.  I  personally  don’t  see  much  use  of 
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Systemigram  for  ATO  or  DEMO  programs.  It  should  be  taught  at  TRADOC  or  Infantry  School 
who  needs  to  generate  ICD  or  ODD 

•  Give  classroom  breaks  at  least  every  90  min.  It’s  difficult  to  concentrate  when  the  session 
becomes  too  long.  It  also  not  good  for  the  health. 

•  Very  comprehensive  -  liked  seeing  certain  principles  reiterated  throughout  material  (i.e.  why 
are  we  practicing  a  skill/completing  a  task?) 

•  Did  not  see  the  overall  value  of  Systemigrams.  I  feel  their  purpose  should  be  explained  in 
more  depth.  The  examples  for  these  in  particular  did  not  seem  to  match  up  well  to  the 
methodology 

•  QFD  -  Great  tool 

•  I  think  more  time  should  be  spent  on  new  concepts  like  the  Systemigram  until  all  participants 
have  seen  ....  of  it  [this  comment  is  hard  to  read] 

•  I  think  a  sample  of  a  good  example  of  each  class  activity  should  be  given  so  that  the 
participants  can  see  if  they  are  missing  anything 

•  Great  course,  recommend  colleagues  to  take  it. 

•  Since  this  is  a  pilot,  timing  of  course  and  content  was  being  assessed  on  the  fly.  Once  fine 
tuned  this  course  will  provide  an  even  better  value.  As  of  now,  the  course  was  valuable. 

•  There  needs  to  be  consistent  definitions  of  all  terms  across  RDECOM,  ARDEC,  and 
TARDEC  call  the  same  concepts  different  things. 
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